N91 - 17 563 1 


^ < ~/~(Sc> 

3 / / ^ ^ • 


MAFT: 

The Multicomputer Architecture for 

Fault-Tolerance 


R. M. KIECKHAFER 

Computer Science and Engineering 
University of Nebraska — Lincoln 
Lincoln, NE 68588-0115 
(402) 472-2402 

rogerk@fergvax.unl.edu 


MAFT is a product of the Allied-Signal Aerospace Company, Columbia MD. 


UNL/CSE/RMK/Augu«tM, 1980 


NASA FM W-SHOP 



Abstract 


'Phis presentation discusses several design decisions made and lessons learned in the 
design of the Multicomputer Architecture for Fault- Tolerance (MAFT). MAFT is a loosely 
coupled multiprocessor system designed to achieve an unreliability of less than 10 ~ 10 /hr in 
flight-critical real-time applications. 

The presentation begins with an overview of the MAFT design objectives and architec- 
ture. It then addresses the fault-tolerant implemention of major system functions in MAFT, 
including Communication, Task Scheduling, Reconfiguration, Clock Synchronization, Data 
Handling and Voting, and Error Handling and Recovery. 

Special attention is given to the need for Byzantine Agreement or Approximate Agree- 
ment in various functions. Different methods were selected to achieve agreement in vari- 
ous subsystems. These methods are illustrated by a more detailed description of the Task 
Scheduling and Error Handling subsystems. 
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Presentation Overview 


• INTRODUCTION 

• SYSTEM FUNCTIONS 

- Communication 

- Task Scheduling 

- Task Reconfiguration 

- Clock Synchronization 

- Data Handling and Voting 

- Error Handling and Recovery 

• SUMMARY 
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Design Objectives 


• RELIABILITY - 1.0 x 10~ 9 over 10 hours. 

• PERFORMANCE 

200 Hz. - Max Task Iteration Rate 
5.5 MIPS - Max Computational Capacity 

1.0 MBPS - Max I/O Transfer Rate 

5.0 ms. - Min Transport Lag (Input — ► Output) 

• REUSABLE 

- Functional Partitioning 

• Application Specific Functions 

• Standard Executive Functions 


• LOW EXECUTIVE OVERHEAD 

- Physical Partitioning 

• Separate Executive Processor 

• Hardware Intensive 
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Loosely-Coupled Multiprocessor 



, INPUT DEV OUTPUT DEV 

• Node => Processor and Private Memory 

• No Shared Memory 

• Message-Based Inter-Node Communication 

• Common Operating System 
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MAFT System Architecture 



SENSORS ACTUATORS 


SYSTEM 

OVERHEAD: 

-COMMUNICATION 

- TASK SCHEDULING 

- RECONFIGURATION 

- DATA VOTING 

- ERROR DETECTION 

- SYNCHRONIZATION 


APPLICATION 

PROGRAMS 


• oc => Operations Controller: 

Special Purpose Device Common to All MAFT Systems. 


• AP => Application Processor: 

General Purpose Application-Specific Processor. 
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Operations Controller Block Diagram 


INTER-NODE INTER-NODE 

MESSAGES IN MESSAGES OUT 



APPLICATION 

PROCESSOR 
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COMMUNICATION 


/ 


UNL/CSE/RMK/Au*u»l 16, 1080 


CS-000 


INTER-PROCESSOR COMMUNICATIONS 


PRIVATE BROADCAST BUS 



I/O DEV 


- INTRA-NETWORK COMMUNICATION 

- MESSAGES TRANSMITTED ON PRIVATE SERIAL BROADCAST BUSSES 

- ALL NODES RECEIVE/ CHECK AND PROCESS ALL MESSAGES 

- MESSAGE TYPES 

- DATA (8/ 16/328 I NT OR BOOL/ IEEE STD 32b FLOAT) 

- TASK COMPLETED / STARTED / BRANCH 

- SYNCHRONIZATION / BRANCH INTERACTIVE CONSISTENCY 

- ERROR REPORT 

- OC / AP COMMUNICATION 

- 1 6 B I T ASYNCHRONOUS P . I . 0 . I NTERFACE 

- LOOKS LIKE "JUST ANOTHER I/O PORT" TO AP 

- COMPATIBLE W EXISTING UNIPROCESSOR OPER SYST 
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Message Handling 


• TRANSMITTER 

- Format Msg - NID f Msg Type, Framing, ECC 

- Broad' ,? * Msg 

• RECEIVERS 1 per incoming link 

- Accent ‘ roperly Framed Bytes 

- Buffer Byte for Message Checker 

• MESSAGE CHECKER 

- Poll Re - 'ers - 6.4 fis cycle 

- Physi nd Logical Checks 

- Steer Good Messages to Other Subsystems 

- Dump Bad Messages into “Bit-Bucket” 
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LOCAL AP/OC INTERFACE OPERATIONS 

1. TASK SWITCHING PROCESS 

- AP: DONE WITH LAST TASK, WHAT IS THE TASK IDENTIFICATION (TID) 

NUMBER OF THE NEXT TASK. 

- OC: HERE IT IS 

2. TRANSFER DATA FROM OC TO AP 

- AP: GIVE ME THE NEXT INPUT DATA VALUE 

- OC: HERE IT IS 

3. TRANSFER DATA FROM AP TO OC 

- AP: HERE ' S THE NEXT OUTPUT DATA VALUE 

- OC: I GOT IT 
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Typical Task System 


PERFORMANCE ISSUES 


• STRICTLY PERIODIC SCHEDULER 

- Fast - Freq Well Above Spec - 500 Hz. vs. 200 Hz. 

- Simple - Binary Freq Dist (/; = 2“*/o) 

- Flexible - Conditional Branching 

- Efficient — Don’t Keep AP Waiting 

• NON-PREEMPTIVE 

- Scheduler Complexity 

- Context Switching Time - Unknown Funct of AP 

- High Frequencies - Short Tasks 

• NO OC INTERRUPTS - I/O 

- Scheduler Complexity 

- Predictability 

- High Frequencies - Polling 

- DMA or IOP access to AP Memory 
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O.C. View of a Task 


• INTERNAL FUNCTION IS BLACK BOX 

• VISIBLE PROPERTIES OF A TASK 

- Priority (static, unique) 

- Iteration Period 

- Precedence Constraints 

- Min and Max duration Limits 

- Fixed Input and Output Shared Data Sets 

- Branch Condition (asserted at completion) 
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FAULT-TOLERANCE ISSUES - I 


• VARIABLE MODULAR REDUNDANCY 

- Specify Redundancy of Each Individual Task 

- Redundancy Matches Criticality 

- No More Copies Than Necessary 


• GLOBAL VERIFICATION 

- Consensus Defines Correctness 

- All Functions Observable and Predictable 

- Replicated Global Scheduler 

- Completed/Started (CS) Message: 

- Node I.D. 

- Started Task I.D. 

- Branch Condition 
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Message Passing Robustness 


• Delivery NOT GUARANTEED 


• Single Msg Error Detect. NOT GUARANTEED 

- ECC coverage > (1 — lx 10 -6 ) per msg 

• Repeated Undet. Errors PROBABILISTICALLY PRE- 
CLUDED 
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TASK SCHEDULING 
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FAULT-TOLERANCE ISSUES - II 


• DISSIMILARITY BETWEEN COPIES 

- Dissimilar Software and Hardware 

- Guards Against Generic Faults 

- No Guarantee - Knight, Levenson, St. Jean 

- Best Chance of Detecting Error 

- Only Chance of Masking Error 

- Implications 

- Different Numerical Results 

- Different Execution Times 

- Impact on Scheduler 

- Min and Max Execution Time Limits 

- Vote on Branch Conditions in CS Messages 
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FAULT-TOLERANCE ISSUES - III 


• BYZANTINE AGREEMENT 

- Definition 

- Agreement on All Messages 

- Validity of Agreement 

- Necessity in MAFT 

- Consensus Defines Correctness 

- Must Have Single Consensus 

- Preconditions for Disagreement 

Initial Disagreement — Enhanced by Dissimilarity 
Assymetric Communication — Minimized by Busses 


Solution Interactive Consistency (Pease et al.) 

- Global Receipt of All Messages 

Periodic Synchronized Re-Broadcast Rounds 

- Vote on Received Re-Broadcasts 

Use Voted Values For All Scheduling Decisions 
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IMPACT OF FAULT-TOLERANCE 


• ALL COPIES DONE BEFORE SUCCESSORS RELEASED 

• MAX EXECUTION TIMERS - ASSURE PROGRESS 

• CONFIRMATION DELAY - MEAN 2.5 SUB. 

- Only Affects Successors 

- Efficiency Requires Parallel Paths 


• FAULT-TOLERANCE LEVELS 

- Single Asymmetric (Byzantine) Fault 

- Double Symmetric Fault 

-Reliability Modelling — 10 -10 //ir with 5 Nodes 
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PERIOD 

SPEC 

DEFINITION 

BOUNDARY 

SUB-ATOMIC 

i 

Min 

400/is 

I.C. Rebroadcast 
Period 

Min Guaranteed 
Task Duration 

Task Inter. Cons. 
(TIC) Message 

ATOMIC 

Min 

2-2.8 ms 

Highest 
Freq. Task 

Clock Sync. 
Period 

System State 
(SS) Message 

' 

GENERAL 

2* 

Intermed. 

System State 

ITERATION 

Atom. Per. 

Freq. Tasks 

(SS) Message 

MASTER 

Max IK 
Atom. Per. 

Lowest 
Freq. Task 

System State 
(SS) Message 


UNL/CSE/RMK/Augu«t 16 , 1990 NASA FM W-SHOP 


PRECEDING PAGE BLANK NOT FILMED 


8 





















Scheduling Stability Problem 


• SCHEDULING INSTABILITY - Anomalous or unpre- 
dictable variations in total execution time (Makespan) 
due to variations in system parameters. 

• MULTIPROCESSOR ANOMALIES - Observation that 
Makespan can be increased by: 

- Increasing Number of Processors, 

- Relaxing Precedence Constraints, 

- Decreasing Individual Task Durations. 


• DYNAMIC FAILURE - Condition where all tasks execute 
properly except that deadlines are missed. 

- Can occur in a fault-free system, 

- Can be induced by instability. 
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Sample Task System 
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Instability of Sample Task System 


• STA NDARD GANTT CHART (max task durations) 


2 4 7 10 


PROC 1 

Ti 

B 

t 4 

? 7 

• 

^ 

• 

PROC 2 

• 1 • 

O 

mm 

T e 

2 
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2 4 6 9 11 


• NO N-STANDARD GANTT CHART (shorten T 3 by e) 


2 4 7 9 


PROC 1 

T x 

T, 

T t 

T s 


PROC 2 

• • « 

3 * 

T e 

T s 

t 7 



2 4 — e 7 — « 9-e 12 - € 


• WH AT HAPPENED? 

- T 3 finished before T 2, 

- T 6 "ready” before T 5( 

- T 5 displaced by Tq =£► Priority Inversion, 

- Critical path (T2 — »• T7) impeded. 
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Previous Work 


• GRAHAM (1969) - Bound Magnitude of Instability 

J 1 

w~ 2 ~N 


- cv = Makespan of Standard Gantt Chart, 

- u/ = Makespan of worst-case schedule, 

- N = Number of Processors. 


• MANACHER (1967) - Stabilization Algorithm 

- Necessary Pre-conditions 

i. 3 “fork" in Precedence Graph, 

ii. Successors of forking task run in parallel on Stan- 
dard Gantt Chart, 

iii. Possible priority inversion around fork. 

- Solution - Impose Artificial Dependency around fork. 
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Stabilized Task System 


• MANACHER ARTIFICIAL DEPENDENCY (T 2 - T 6 ) 



• EFFECT 

- T 2 is common parent for both X5 and T&, 

- T 6 will be "ready" no earlier than T 5t 

- T 5 precedes Tq in priority list, 

- T 6 can not be selected before T5. 
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Limitations of IVlanacher’s Solution 


• Sufficient, but not always necessary 

• Adds Scheduling Overhead (resolve edge) 

• Unrealistic System Model 

- Assumes no scheduler overhead, 

- Assumes dynamic allocation, 

- Allows for no Confirmation Delay, 

- Ignores minimum duration bounds, 

- Does not predict magnitude of instability. 
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Current Research 


• Find Necessary and Sufficient Stability Conditions. 

• Develop Stabilization Strategies 

- Task System Stabilization 

• Edge Stabilization (Manacher) 

• Vertex Stabilization 

• Hybrid Stabilization 

- Run-Time Scheduler Stabilization 

• Limited Scan Depth 

- Scheduling Algorithm Stabilization 

• Sched. Algorithm Assigns Priorities 

• Constrain to Preclude Necessary Conditions 

• Extend System Environment 

- Scheduler Overhead 

- Static Allocation 

- Confirmation Delay 

- Minimum Duration Bounds 
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SYNCHRONIZATION 
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MAFT Synchronization 


• Periodically Exchange System State (SS) Msgs 

- SS Msg => “Atomic Period" Boundary 

- Synchronization Period = 2 Atomic Periods 

• Loosely Synchronized Individual Clocks 

- Msg Exchange => No Separate Clock Lines 

- Physical Separation => Damage Tolerance 

- Robustness to “Common Upset" events 

• Synchronization Modes 

- Steady State - Maintain Existing Synchronization 

- Warm Start - Converge to Existing Operating Set 

- Cold Start - Form Initial Operating Set 

• Interactive Convergence to synchronize 

• Interactive Consistency Steady State 

• Origin of Two-phase algorithm 
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DATA HANDLING AND VOTING 
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Typical Sync. Values 


• 6 = 7 iisec - 600 ft. separation 

• p = 5 • 10~ 5 

• R = 20 msec => 10 msec Atomic Pd. => 100 Hz. 

• pi? = 1 /zsec 

• No Faults: Max <5 = 8.5/x sec 

• With Faults: Max 8 = 16.5/x sec 
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Data Management 


• DATA GENERATED BY AP 

• BROADCAST IN DATA MESSAGE 

• RECEIVED AND PROCESSED BY ALL NDOES 

- Static Limit Check 

- On-The-Fly Vote 

- Dynamic Deviance Check 
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On-The-Fly Voting I 


• TRIGGERED BY DATA MESSAGE ARRIVAL 

• DATA ID ACTS AS UNIQUE VARIABLE NAME 

• USE ALL PREVIOUS COPIES OF SAME DATA ID 

- MS or MME (programmer selectable) 

• Sort Serially - High-Order-Bit First 

• Select 2 “Medial” Values 

• Average (Add and Shift) 

- No I.C. Vote for Boolean Types 

• Difficult to implelement round 2 

• Usually Control Data for Mode Switch 

• 3 Better Way for Mode Switch 
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On- The- Fly Voting II 


• DEVIANCE CHECK 

- Compare Each Copy to Voted Value 

- Excessive Difference error 

- Programmer Sets Limits 

- Generate Error Vector => Source Nodes 

• TERMINATE 

- Scheduler Says All Copies Done 

- Send Error Vector to Fault-Tolerator 

- Send Voted Value to Data Memory 

- Swap On-line/Off-line Buffers in Data Memory 

- Clear Previously Received Copies from Voter 
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ERROR HANDLING AND RECOCVERY 


-r 
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Fault Classifications 


• BYZANTINE (MALICIOUS) 

Pease et al. (1982) 

- N > 3t + 1 

- r > t 

• MALICIOUS u BENIGN (self-evident) 

Meyer and Pradhan (1987) 

- t = m + b 

- N > 3m + b + 1 

- r > m 

• (ASYMMETRIC u SYMMETRIC) u BENIGN 

Tliambidurai and Park (1989) 

- t = CL S b 

-N>3a + 2s + b + r + l 

- r > a 
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Fault Classes by Source 



• Can Estimate Separate A’s 


- A 

- A 


asym 

sym 






10" 6 

1(T 3 . . . 1(T 4 


• Generic Fault = Multiple Symmetric 




? 
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Error Detection 


• Errors Are Manifested In Messages 

- Physical: ECC, framing, length 

- Contents: values 

- Timing or sequencing 

- Existence or non-existence 

• Log Errors Over One Atomic Period 

- Errors reported by all subsystems 

- Fault-Tolerator records errors 

- 3 31 separate error "flags" 

- 3 Unique "Penalty Weight" PW for each flag 

- 3 "Incremental Penalty Count” IPC for each node 

- FOR each flag / reported against node i : 

• IPC(i) := IPC(i) + PW(f) 
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Error Reporting 


• Broadcast ERR(z) Message 

- At beginning of next Atomic Period 

- Contents: 

• IPC(i ) 

• BPC{i ) - Base (current) penalty count 

• All Error Flags for node i 

• No ERR Message => No Detections 
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BPC Manipulation 


• BPC =* Health Of Node 

• Increasing BPC - ERR Message Vote 

- Vote on BPC(i) 

- Vote on IPC(i) 

- BPC(i) := BPC{i) 4- IPC(i ) 

• Decreasing BPC - Fixed decrement 

- 3 Penalty Decrement value PD 

- At New Master Period 

- BPC(*) := BPC(*) - PD 

- Allows For Eventual Readmission 
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Exclusion/Readmission 


• Recommend Exclusion/Readmission 

- 3 Exclusion Threshold T exc \ 

- 3 Admission Threshold T adm 

- Recommend in next SS message: 

• BPC{i) > T exc i => Exclude i 

• BPC[i ) < T adm =» Readmit i 

• T adm < BPC(i) < T exc i => No Change 

• I.C. Vote on Recommendations 

- Consistent System State is Critical 

- Free (needed for cold-start) 

- Highly Degraded Systems 

- Common Mode Upset Recovery 
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Sed Quis Custodiet . . . Ill 


• AP - Diagnostics in Workload 

• OC - System Level Self-Test 

- Errors Very Rare 

- Inject Faults to Excercise Error Detection 

• Special self-test Task ID 

• Suspend normal Transmitter Ops 

• Tranmsit string from self-test ROM 

• Can transmit ANY test scenario 

- Test Results Based On 

• False/Missed Accusations 

• Cyclic Link Check 

- Independent of Actual Bit-Stream 

- Rotate "Originator” Duty 

- Complete Coverage If ANY One Node Correct 
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Version Management 


• SSV = System State Vec - eg (2,1,1) 

• VMV = Version Management Vec - eg (1,1,1) 

• WMV = Workload Management Vec — (SSV) or (VMV) 

• Vectors Used By Different Subsystems 

Data Voter VMV Inactive Copy Ignored For Vote 
Dev Checker SSV Inactive Copy Still Monitored 
Scheduler WMV Inactive Copy May Not Run 


• WMV = SSV 

- Inactive Copy Still Executing 

- Actual Tasks Being Monitored 

- Best for Generic Fault Detection 

• WMV = VMV 

- Inactive Copy Doing Something Else 

- Will Not Be Affected By Generic 

- Can Activate To Replace Sibling 

- Best For Generic Recovery 
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Synchronizer Error Detection 


• MAFT error detection is by consensus 

- Each node reports errors on all nodes. 

- Majority vote confirms or denies accusations. 

- Disagreement with majority may itself be an error. 

• Faulty node must be detected by majority of nodes 

- Must be "far enough" out of sync 

- There exists a region of ambiguity 

- Defines size of "Sync Window” 
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Synchronizer Error Windows 
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• W s = SOFT ERROR WINDOW 

- Spans Range of Receipts from Non-Faulty Nodes 

- Error May Not Be Confirmed 

- Inherent Ambiguity 

- Must Suspend Error Disagreement Penalties 

• W h = HARD ERROR WINDOW 

- IF Any non-faulty node detects a Hard-Error 
THEN All non-faulty nodes detect an Error 

- Can demand Corroboration 
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Typical Sync. Window Values 


• e = 7 nsec - 600 ft. separation 

• p = 5 • 10~ 5 

• R = 20 msec => 10 msec Atomic Pd. => 100 Hz. 

• pR = 1 psec 

• No Faults: Max 6 = S.bp sec 

• With Faults: Max 6 — 16.5/z sec 

• W s = AQp sec 

• Wf t = 87 p, sec 
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SUMMARY 


gr 
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SUMMARY COMMENTS ON THE APPLICATION OF MAFT TECHNOLOGY 


1. CAPABILITIES 

- BASIS OF A GENERIC REAL-TIME MULTICOMPUTER SYSTEM 

- REMOVES F.T. OVERHEAD FROM APPLICATION PROCESSOR 

- HANDLES ALL REDUNDANCY MANAGEMENT WITHIN COMPUTER 

- ASSISTS IN REDUNDANCY MANAGEMENT OF I/O SYSTEM 

2. FLEXIBILITY 

- INDEPENDENT OF I/O ARCHITECTURE 

- HIGHLY RECONFIGURABLE AND GRACEFULLY DEGRADABLE 

- PROVIDES MECHANISMS/ NOT POLICIES 

3. USABILITY 
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ADVANTAGES OF APPROACH 


- PARTITIONED APPROACH SIGNIFICANTLY REDUCES PROCESSOR OVERHEAD 

- DATA DRIVEN ARCHITECTURE MUCH FASTER THAN SOFTWARE IMPLEMENTATION 

- NOT DEPENDENT UPON ARCHITECTURE OF APPLICATION PROCESSOR 

- REDUNDANCY IS "TASK-BASED" AND FLEXIBLE 

- SUITABLE FOR HIGH RELIABILITY AND HIGH PERFORMANCE APPLICATIONS 
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